A note on adoption: two patterns strict mode will break, and what that tells us we're missing

I've spent time with this PR and the spec, and I want to say up front: the provenance trinity is a real evolution for DataJoint. strict_provenance is exactly the right mechanism to turn "reads should come from declared ancestors" from a convention we hope people follow into a guarantee the framework actually enforces. None of what follows is a blocker, and none of it is a criticism of the design.

What I want to do here is think through adoption. There are a couple of patterns that are fairly commonly used in practice today that this feature will break. Both of them are, honestly, bad patterns — they're provenance bugs, and strict mode is right to catch them. But they're bad patterns that a lot of working pipelines lean on, and I think if we turn this on without a story for how those pipelines migrate, we take DataJoint to the next level and leave a chunk of our users behind. So this is really an argument for the next thing we should build, using these two patterns to show why.

Pattern 1 — reading "sideways" to a table that isn't a declared ancestor

The common case looks like this: a computation needs a piece of reference information that lives off to the side of its own lineage. For example, SpikeDetection depends on Recording, but to set a baseline it wants the brain region — which it gets by walking up to Subject and then back down a different branch to Surgery → BrainRegion:

class SpikeDetection(dj.Computed):
    definition = "-> Recording"
    def make(self, key):
        signal = (Recording & key).fetch1("signal")          # declared ancestor — fine
        region = (BrainRegion & (Surgery & key)).fetch1("region")  # NOT an ancestor
        baseline = BASELINES[region]
        ...

BrainRegion isn't upstream of SpikeDetection — it's a cousin. Under strict mode this read is blocked, and that's correct: if the region's baseline feeds the result, then BrainRegion genuinely is a dependency, and today nothing records that. Edit a BrainRegion row and every SpikeDetection derived from it is silently stale, with no way to know. So strict mode is doing exactly its job — turning a silent, invisible dependency into a loud, declared one.

Worth noting the boundary: the very common "parameter table" case (-> AnalysisParams) is already a declared ancestor, so it keeps working. Only genuinely undeclared sideways reads break. This one has a reasonably clear fix — make the dependency real by declaring it — so I'll spend most of the space on the second pattern, which is harder.

Pattern 2 — one make() that ingests a file and writes to many tables

This one is about using DataJoint's populate machinery (job reservation, error tracking, triggering on new data) as an orchestration layer. A very common shape: a table triggers when a new file appears, and its make() parses that file — which usually contains nested, hierarchical information — and fans the results out into a bunch of normalized entity tables:

class FileIngestion(dj.Computed):
    definition = "-> FileManifest"
    def make(self, key):
        data = parse(fetch_file(key))
        Session.insert(data["sessions"])   # writing to tables that aren't self
        Block.insert(data["blocks"])
        Trial.insert(data["trials"])
        Event.insert(data["events"])

Strict mode blocks this, because a make() may only write to self and its Parts. And again it's right to: those Session/Trial/Event rows land with no foreign-key edge back to the file they came from, so there's no way to trace an entity to its source. The traceability we care about is genuinely broken here.

The obvious suggestion is "make those tables Parts of FileIngestion." That doesn't work, and the reason is instructive: Session, Trial, and Event are top-level entities that other pipeline branches depend on (Recording -> Session, SpikeSorting -> Trial, and so on). Turning them into Parts of an ingestion table would force all those downstream branches to reference a Part from outside its master — the exact thing the master/Part convention tells us not to do. So Parts are structurally the wrong tool here, not just semantically awkward.

The hard part: for pattern 2 there's no clean replacement, and here's why

When I work through the alternatives for orchestrated ingestion, every strict-compliant option gives up something real. There are three:

Option A — the "landing-table" approach. Restructure so the ingestion is a real upstream table: ParsedFile becomes an ancestor, and Session, Trial, etc. each depend on it and get their own single-target make() that reads the already-parsed data and writes only to itself. This is strict-compliant and provenance is intact. But it means your entities are now downstream of a file. The pipeline reads as "everything begins with a data source" rather than "everything begins with a subject and a session." It's input-driven design, and I don't think it reflects the truth of the model.

Option B — keep the entities as roots, orchestrate outside DataJoint. Session etc. stay top-level tables, and some external system (a script, Airflow, whatever) does the file-watching and inserts them. The domain model stays clean — but you lose the DataJoint populate machinery you wanted in the first place, and the file→entity link now lives entirely outside DataJoint, invisible to it.

Option C — the fat make() above. Clean domain model, keeps orchestration — but it's the pattern strict mode exists to forbid, precisely because it severs provenance.

Laid out together:

Every row loses at least one thing that matters.

Why the "landing table" option is worse than it first looks

Option A is the one people will reach for, so it's worth being clear about its deepest problem: it bakes a transient operational fact into the durable data model.

How an entity physically arrives is not durable. A Session might come from a file today, an API call next quarter, a direct database sync the year after. But Session belongs to Subject — that's the real, invariant truth of the model, and it holds no matter how the data got there. These two facts live on completely different timescales: the domain relationships change on the scale of scientific redesign (years), while ingestion mechanics change on the scale of infrastructure (months).

The foreign-key graph is supposed to encode the durable, invariant relationships. The moment you put "came from this file" into it, you've married the slow-moving graph to the fast-moving one. Every time ingestion changes, you're doing a schema migration; and your historical rows permanently carry whatever ingestion ancestor happened to exist when they were created. So the principled strict-mode option is actually the least stable one over the life of a pipeline — enforcing provenance this way makes the schema churn with your infrastructure. That's a genuinely counterintuitive result, and I think it's the crux.

Off versus on, stated carefully

Here's the asymmetry that worries me. With strict_provenance off, a workable option exists — the fat make() — that keeps a clean, durable domain model and keeps DataJoint orchestration. Its only casualty is provenance, which the framework wasn't guaranteeing anyway. With it on, that option is gone, and there's no remaining option that records the file→entity relationship truthfully — as "how it arrived," rather than as a fabricated (and non-durable) dependency in the FK graph — while keeping orchestration. So turning strict mode on doesn't just close a loophole; for this class of pipeline it takes away the one option whose only cost was something already soft, and leaves you choosing between corrupting the durable model (A) or giving up the framework's orchestration (B).

What this actually points to — two things worth building

I don't think the takeaway is "don't ship strict mode." I think it's that strict mode is the feature that makes two follow-on capabilities necessary — it's the alarm, and these are the fire exits:

1. Provenance-first schema evolution. The reason both patterns are so painful to fix is that the remediation is always "restructure your schema," and we have no tooling for that — so people fall back on drop-and-repopulate, which destroys history. A proper schema-evolution tool would use the trinity's own primitives — trace to find what's upstream of a change, cascade to find what it invalidates — to change dependency structure and recompute safely. It's not alembic; it's evolution that manages the invalidation cascade. And it's built from exactly what this PR ships, which is what makes it a natural next step rather than a detour.

2. A first-class notion of where data came from, kept separate from the FK graph. Everything above keeps circling one gap: DataJoint has one graph, and it uses it for both "what is derived from what" (durable, invariant) and, when forced, "how this row arrived" (transient, operational). Those need to be different things. Ingestion lineage should be recorded — append-only, allowing many sources over time — so a Session can carry a "came from file X" record today and an "came from API Y" record later, with no schema change at all. The FK graph structurally can't do that, and forcing it to (option A) is exactly what corrupts the model.

Neither of these is a 2.3 concern, and I'm not suggesting they gate this release. But I'd love for us to decide the sequencing deliberately, so we're not shipping the alarm well ahead of the exits. Happy to write these up properly as their own proposals if that's useful.